Inaccessibility in online learning of recurrent neural networks.

We apply nonlinear dynamical system techniques to recurrent neural networks. In particular, we numerically analyze the dynamical system characteristics of the online learning process. By introducing the notion of inaccessibility, we show that the learning process is well characterized by strong nonhyperbolicity and inaccessibility, which is a greater uncertainty than chaotic unpredictability. These results are clearly contrasted with a gradient descent dynamics, or ordinary chaos.